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Intellectual Property Rights 



IPRs essential or potentially essential to the present document may have been declared to ETSI. The information 
pertaining to these essential IPRs, if any, is publicly available for ETSI members and non-members, and can be found 
in ETSI SR 000 314: "Intellectual Property Rights (IPRs); Essential, or potentially Essential, IPRs notified to ETSI in 
respect of ETSI standards", which is available from the ETSI Secretariat. Latest updates are available on the ETSI Web 
server ( http://webapp.etsi.org/IPR/home.asp ). 

Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee 
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web 
server) which are, or may be, or may become, essential to the present document. 



Foreword 

This Technical Specification (TS) has been produced by ETSI Technical Committee Telecommunications and Internet 
converged Services and Protocols for Advanced Networking (TISPAN). 

The present document is part 1 of a multi-part deliverable covering the IMS/PES Performance Benchmark, as identified 
below: 

Part 1: "Core Concepts"; 

Part 2: "Subsystem Configurations and Benchmarks"; 

Pai-t 3 : "Traffic Sets and Traffic Profiles" ; 

Part 4: "Reference Load network quality parameters". 
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Scope 



The present document is for an initial release of a PSTN/ISDN Emulation Sub-system (PES) performance benchmark. 
The same tests can be used also for legacy PSTN/ISDN networks or for inter-working tests between PSTN/ISDN 
emulation subsystem and legacy PSTN and ISDN. The metrics measured and reported are for performance of this 
subsystem under a communications application load. The present document is the first part of the multi-part deliverable 
which consists of four parts. 

The present document contains the overall benchmark descriptions, architectures, processes, and information 
models that are common to all specific benchmarking scenarios. 

TS 186 025-2 [1] contains the specific benchmarking use-cases and scenarios, along with scenario specific metrics and 
design objectives. It also defines the SUT configuration parameters. This part also contains any required extensions to 
the overall descriptions present in the present document, if necessary for the specific scenario. 

TS 186 025-3 [i.l] defines an initial benchmark test through the specification of a traffic set, traffic-time profile and 
benchmark test procedure. 

TS 186 025-4 [i.2] defines Reference Load network quality parameters for the use cases defined in TS 186 025-2 [1]. 



2 References 

References are either specific (identified by date of publication and/or edition number or version number) or 
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the 
reference document (including any amendments) applies. 

Referenced documents which are not found to be publicly available in the expected location might be found at 
http://docbox.etsi.org/Reference . 

NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee 
their long term validity. 

2.1 Normative references 

The following referenced documents are necessary for the application of the present document. 

[1] ETSI TS 186 025-2: "Telecommunications and Internet converged Services and Protocols for 

Advanced Networking (TISPAN); IMS/PES Performance Benchmark Part 2: Subsystem 
Configurations and Benchmarks". 

[2] ITU-T Recommendation Q.543: "Digital exchange performance design objective". 

2.2 Informative references 

The following referenced documents are not necessary for the application of the present document but they assist the 
user with regard to a particular subject area. 

[i.l] ETSI TS 186 025-3: "Telecommunications and Internet Converged Services and Protocols for 

Advanced Networking (TISPAN); IMS/PES Performance Benchmark; Part 3: Traffic Sets and 
Traffic Profiles". 

[i.2] ETSI TS 186 025-4: "Telecommunications and Internet Converged Services and Protocols for 

Advanced Networking (TISPAN); IMS/PES Performance Benchmark; Part 4: Reference Load 
network quality parameters". 

[i.3] ITU-T Recommendation P. 862: "Perceptual evaluation of speech quality (PESQ): An objective 

method for end-to-end speech quality assessment of narrow-band telephone networks and speech 
codecs". 
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[i.4] ITU-T Recommendation P. 862. 1: "Mapping function for transforming P. 862 raw result scores to 

MOS-LQO". 

[i.5] ITU-T Recommendation P. 56: "Objective measurement of active speech level". 



3 Definitions and abbreviations 

3.1 Definitions 

For the purposes of the present document, the following terms and definitions apply: 

background load: workload applied to an SUT during a benchmark test, for the purpose of consuming SUT resources 
during a benchmark test and changing the traffic intensity at which the capacity of the SUT is reached 

benchmark report: document generated at the conclusion of a test procedure containing the metrics measured during 
the execution of the test and/or computed from the data collected in the benchmark log 

benchmark test: procedure by which a test system interacts with a System Under Test to measure its behaviour and 
produce a benchmark report 

configuration: specification of a subset of IMS/PES architectural elements and metrics for which collection of 
benchmark tests can be defined 

design objective: probabilistic model of delay and failure requirements for SUT, associated with a use-case, specified 
by threshold values and probabilities for delay and scenario failure 

idle load: load that is not dependent on the traffic or other external activities 

maximum capacity: maximum processor load that a processor can handle without rejecting new calls 

metric: performance measurement of SUT reported in a benchmark report 

parameter: attribute of a SUT, test system, system load, or traffic set whose value is set externally and prior to a 
benchmark test, and whose value affects the behaviour of the benchmark test 

processor load: part of time the processor executes work, normally expressed in percent 

NOTE: The processor load consists of Idle load. Traffic load and Usage load. 

Reference Call (RC): basic ISUP to ISUP call connected through two MOW in the same MGC domain 

test parameters: parameters whose values determine the behaviour of a benchmark test 

test procedure: specification of the steps to be performed by a benchmark test 

test scenario: specific path through a use-case, whose implementation by a test system creates a system load 

test system: collection of hardware and software which presents a system load to a system under test and collects data 
on the system under test's performance, from which metrics can be computed 

traffic load: load that results from handling traffic events that are directly related to calls 

NOTE: This load varies with the traffic intensity. 
traffic-time profile: evolution of the average scenario over a time interval 
traffic set: mixture of traffic scenarios 

usage load: load that is reserved for the administrations operation and maintenance activities during busy hour 
workload: number of reference calls per second (RC/s) 

NOTE: It is calculated by multiplying calls per second by its corresponding WLF. 
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workload factor (WLF): traffic load for different types of calls in relation to the traffic load of the reference call (ISUP 
call) 



3.2 



Abbreviations 



For the purposes of the present document, the following abbreviations apply: 

%IHS Percent Inadequately Handled Scenarios 

A-BGF Access Border Gateway Function 

AGCF Access Gateway Control Function 

APS Attempts per Second 

EC Bearer Capability 

CLIP Calling Line Identification Presentation 

CW Communication Waiting 

DO Design Objective 

IMS IP Multimedia Subsystem 

ISDN Integrated Service Digital Network 

ISUP ISDN User Part 

MCID Malicious Communication IDentification 

MGW Media Gateway 

MHT Mean Holding Time 

NGN Next Generation Networks 

PIXIT Protocol Implementation eXtra Information for Testing 

PSTN Public Switched Telecommunications Network 

RC Reference Call 

SAPS Session Attempts Per Second 

SUT System Under Test 

TA Tones and Announcement 

TSS Telephony Softswitch Solution 

UDI Unrestricted Digital Information 

WLF WorkLoad Factor 



Benchmark information model 



In this clause, "benchmark information model" refers to the structure of the information elements that define the 
benchmark. This information model is depicted in figure 1 . 
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Figure 1 : IIVIS/PES benchmark information model 
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The information model consists of three primary elements: use-cases, which describe the behaviour of an individual 
user, and which in turn define scenarios; benchmark tests, which generate a workload by aggregating the behaviour of 
individual scenarios in a controlled manner, and collect log files of measurements during the test; and benchmark test 
reports, which report metrics interpreted from the benchmark test log files. 

4.1 Use-case 

The top level of the individual behavioural model is the use-case. A use-case describes the goal that a user has in 
interacting with a system, the various actors (e.g. other users, network elements) that participate in the use-case, the 
basic course of events that are carried out by the user and the SUT, the design objective of the use-case, the possible 
outcomes that apply to the use-case, and the metrics to be collected. The goal and actors of a use-case are documented 
in narrative text and diagrams; the other elements are complex information elements, which are described in their 
respective clauses. 

4.1.1 Call Flow 

The calls flows define the characteristic message flows, the tones and announcement for a specific interface. 

4.1.2 Load Profile 

To facilitate the calculation of processing capacity and the appropriate load profile the concept of workload factor has 
been defined based on the reference call for each combination of traffic case and traffic signalling interface. The 
reference call (RC) is defined as a basic ISUP to ISUP call connected through two MGW in the same domain. 

Based on the workload factors for all different types of calls, the call intensities and the services used, one can express 
the total trafflc load in an equivalent number of reference calls per second. 

The dimensioning of any type of network depends on a number of different parameters such as utilization per channel, 
calls per second, mean holding time, type of accesses being involved, and type of services being requested. 

4.1.3 Metrics 

The metrics of a use-case describe the measurements collected from the execution of a scenario attempt. Typical 
metrics include response times and message rates. If a scenario is selected for execution in a benchmark test, its metrics 
are collected. See clause 7 for more detail. 

4.1 .4 Use-case outcomes 

A use-case outcome is a set of possible outcomes of the scenarios of a use-case. An outcome may be simply "correct", it 
may reflect an error or failure condition; or it may reflect a correct behaviour that took an excessive amount of time to 
occur. An instance of a scenario that experiences an error, failure, or timeout outcome is referred to as an inadequately 
handled scenario attempt. 

4.1 .5 Scenarios and scenario attempts 

A scenario is a trace of a path through a use-case. It is analogous to "call attempt", but applies to all interactions within 
an IMS/PES network, different Bearer, and application interactions. 

A scenario may succeed, fail, or succeed functionally. 

The terms "scenario attempt" and "scenario attempts per second" are used in this standard in place of "call attempt" and 
"call attempts per second" because IMS/PES is a transaction-oriented system with transactions of a variety of types 
(e.g. speech calls, 3,1 kHz calls, modem calls. Fax calls, etc.). Traffic sets, and indeed the real world, do not operate 
according to only one transaction type, so the more generalized term is necessary. It would be incorrect and misleading 
to attempt to report the capacity of a system in "call attempts per second", "registration attempts per second", etc., for 
system loads that were other than purely call attempts, registration attempts, etc. 
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4.1 .6 Design Objective (DO) 

The Design Objective (DO) describes the maximal acceptable rate handled scenario attempts for a use-case. 

4.1.7 Scenario 

A scenario describes a single interaction sequence among the actors of a use-case. It is documented by a set of 
preconditions on its actors (typically specified by parameter values). In case of IMS/PES the scenario is defined as a set 
of different Bearer Capabilities (e.g. speech , 3,1 kHz audio, UDI, UDI/TA), services (fax, modem ) or dial mode. 

4.2 Benciimark test 

A benchmark by definition measures the behaviour of a population of users. To accomplish this, the behaviours of 
individual users must be aggregated into input traffic to the SUT. The input traffic must be realistic, in the sense that a 
population of users would perform such actions in the real world, and in the sense that statistical variation in user 
behaviour is similar to statistical variation that would occur in the real world. 

4.2.1 Traffic set 

The traffic set is a collection of scenarios which are determined to be likely to co-occur in a real-world scenario. The 
scenarios do not need to come from the same use-case. Within a traffic set, each scenario has an associated relative 
occurrence frequency, interpreted as the probability with which it would occur in the course of the test procedure. 

4.2.2 Background load 

Background load is a workload presented to the SUT in order to consume its resources. It may consist of a stream of 
traffic presented to the SUT by an external system apart from the test system; or it may be a workload presented to the 
processing elements, network, or storage subsystem of the SUT. 

The purpose of background traffic is to make possible the measurement of a design objective capacity in SUT when the 
capacity of the test system is insufficient to reach the design objective capacity. 

If a benchmark test is published in which background load is used, then the following requirements apply: 

• The hardware used to generate the background load must be fully specified. If the background load is 
generated by software running directly on the SUT, then the components of the SUT on which the background 
load is executed must be fully specified. 

• The software used to generate the background load must be provided in source form, including make files and 
any other configuration files required to compile the software. 

4.2.3 Traffic-time profile 

The traffic-time profile is a function describing the average scenario attempt arrival rate as a function of elapsed time 
during a benchmark test. A traffic-time profile should be chosen in such a manner that, for a given scenario attempt 
arrival rate, sufficient samples are generated that metrics can be collected with an appropriate confidence bound. 
Following Call Profiler Traffic Patterns are used today: Saw Tooth Blast Ramp Steady Call Rate Rolling Blast Poisson 
Distribution. To get a realistic scenario a combination of at least two scenarios is needed. 



4.2.4 Test parameters 



The benchmark test parameters are used to control the behaviour of the test script. The data elements required to 
configure the test system are listed in table 1 . 

Table 1 is a non-exhaustive list of test parameters defined for the benchmark standard. The list is expected to grow over 
time, as additional subsystems and system configurations are developed. 
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Table 1 : Test parameters 



Parameter 


Description 


Start time 


Amount of time that a system load is presented to a SUT at 
the start of a test 


Stop time 


Amount of time that a system load is presented to a SUT at 
the end of a test 


TotalProvisionedSubscribers 


The number of simulated subscribers provisioned in the 
network 


PercentSimulatedSubscriber 


The average percentage of simulated subscribers 


Simulated Maximum simultaneous call legs 


The number of simulated Maximum simultaneous call legs 


Traffic per subscriber 


Traffic per subscriber; default value 0,1 Eriang 


MHT 


Mean Holding Time of a call ; default value 120 seconds 


Ringing time 


Duration between (180 ringing and 200 OK INVITE ) 
Default value 1 to 5 s 


NoS 


number of subscribers originating traffic per subscriber 


CAPS/BHCA 


Call attempts per second / busy hour call attempts 


WLF for Call Controller 


The workload factor for Call Controller for specific 
configuration. Default value 1 to 3 


WLF for Gateway Controller 


The workload factor for Gateway Controller for specific 
configuration. Default value 1 to 3 


WLF for MGW 


The workload factor for Media Gateways for specific 
configuration. Default value 1 to 3 


TDM Trunks 


Number of TDM trunks 


ETH 


Number of ETH Connections 


Type of call 


Calls requesting DTMF receiver 

POTS calls with CLIP 

POTS calls with CW 

POTS calls with Call diversion 

POTS calls with MOID 

POTS calls with Fax 

POTS calls with Modem 

IN calls using GS devices 

Calls with tone sending for Continuity Check 

Calls requesting a loop for Continuity Check 

ISDN calls with CLIP 

ISDN calls with CW 

ISDN calls with Call diversion 

ISDN Call with MCID 

ISDN calls with Fax 

ISDN calls with Modem 

ISDN with BC=3.1 kHz 

ISDN with BC= speech 

ISDN with BC=UDI 


Protocol call type 


SIP-I 
Basic 
PRACK 
PRAC & PREC 


SIP 
Basic 
PRACK 
PRAC & PREC 


Protocol call type 


H.323 

Fast connect 

Tunnelling 

Separate H.245 


SIGTRAN 
M3UA(ISUP) 
lUA/DUA 
M2PA 


DNS/ENUM 


ISUP routes utilization usually 


ISUP routes utilization usually, Default value 0,8 



£75/ 



12 



ETSI TS 186 025-1 V2.1.1 (2011-01) 



Parameter 


Description 


Traffic Case 


MGW (ISUP) - MGW (ISUP) • inter-MGW 

MGW (POTS) - MGW (POTS) • inter-MGW 

MGW (BA/PRA) - MGW (BA/PRA) • inter-MGW 

MGW (PRA - MGW (PRA) • inter-MGW 

MGW (POTS) - MGW (BA/PRA) • inter-MGW 

MGW (V5.2 POTS) - MGW (V5.2 POTS) • inter-MGW 

MGW (ISUP) - MGW (ISUP) • intra-MGW 

MGW (POTS) - MGW (POTS) • intra-MGW 

MGW (BA/PRA) - MGW (BA/PRA) • intra-MGW 

MGW (PRA) - MGW (PRA) • intra-MGW 

MGW (V5.2 POTS) - MGW (V5.2 POTS) • intra-MGW 

MGW (ISUP) -AGW (POTS) 

MGW (ISUP) -AGW (ISDN) 

MGW (ISUP) -SIP-I 

MGW (POTS) - SIP-I 

MGW (ISDN) -SIP-I 

MGW (PRA) - SIP-I 

MGW (V5.2 POTS) - SIP-I 

SIP -SIP Transit 

H.323 - H.323 

SIP-H.323 



4.3 Benchmark report 



A test report is a document, with accompanying data files, that provides a full description of an execution of a 
benchmark test on a test system. The SUT and test system, as well as their parameters, are described in sufficient detail 
that an independent test site can replicate the test. The results of the test include data, represented as charts and data 
sets, depicting the behaviour of the SUT over the elapsed time of the test; reports of the relevant metrics that are 
conventionally used to compare benchmark results of differing SUTs; and a full description of other observations and 
exceptions noted during the test. 



System Under Test (SUT) 



The IMS/PES performance benchmark covers benchmark tests for the PSTN/ISDN Emulation Sub-system (PES), The 
same tests can be used also for legacy PSTN/ISDN networks or for inter-working tests between PSTN/ISDN emulation 
subsystem and legacy PSTN and ISDN. 

The following functional entities appear to be necessary from the perspective of specifying information flows and 
ensuring the interoperability of services: 

Access Gateway Analogue line function 

Access Gateway BRI function 

Access Gateway PRI function 

Residential Gateway Analogue line function 

Residential Gateway BRI function 

Trunk Gateway function 

Access Call Server function 

Transit Call Server function 

Packet Handler Gateway function 

Media Gateway Controller function 

Media Server Control Function 
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• Customer Location function 

• IN Access Subsystem 

• SIP Server Access Function 

• Trunk Signalling Gateway 

The Functional Architecture is shown in figure 2 in such a way that it can be seen that multiple implementation 
architectures are possible. There are some fundamental points that should not be missed however. The first of these is 
that we have gateways that convert legacy interfaces such as national analogue PSTN Z reference points and ISDN S or 
T reference points into NGN interfaces. These are usually thought of as being H.248 interfaces but that is not the only 
interface that can be used. Depending on the service set MGCP or interfaces carrying suitable information in SIP can be 
used. The key point is that the information flow can carry the stimulus information traditionally needed in national 
PSTNs to carry both line and register signalling from customers as well as speciaUsed service signalling. 



NOTE: 

Lines inside the grey area are purely illustrative. 

Information flows may 
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Figure 2: Overview of Functional Entities 
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Figure 3: AGCF/VGW session processing model 



5.1 



Creation of subscriber data base 



The subscriber data base is the data-set required to configure the SUT in order to execute a benchmark test. Using the 
same data, the test system should be able to generate correct traffic. 

The present document does not try to specify a complete set, but rather just the subset that will ensure comparable 
results. All other provisioning information required for correct configuration of a SUT is to be set at the discretion of 
the SUT provider. 

One requirement for a fair benchmark is that the input data is similar for all test-runs. In order for this to happen we 
have two choices: 

a) Provide data base for the subscriber base. However, because we have to ensure scalability for the benchmark, 
this solution is not feasible. 

b) Provide rules to generate this data and data generators. Algorithms using random generators will be avoided 
for data that could possibly influence the results. 



Test system 



The test system is used to generate the appropriate load on the SUT. The present document does not mandate any 
specific test system to be used, although the details of the test system must be reported in the benchmark report. 

The test system should have two main functions: 

• Traffic generation: the test system must be able to execute use-cases' scenarios following the traffic-time 
profile. It must also be able to reproduce the appropriate traffic set (a mix of scenarios with a weight for each 
of them). 

• Network emulation: optionally, network characteristics on the different interfaces should be emulated by the 
test system. 
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7 



Benchmark metrics examples 



The metrics reported by a benchmark test is measured in real time during the execution of the test, or may be computed 
after completion of the test from event logs collected during the execution. Enclosed is a benchmark matrix based on 
the ITU-T Recommendation Q.543 [2]. 

Table 2: Benchmark metrics examples 



delay parameters 


Description 


call request delay 


For ANALOGUE SUBSCRIBER 
LINES, call request delay is defined as 
the interval from the instant when the 
off-hook condition is recognizable at 
the subscriber line interface of the 
exchange until the exchange begins to 
apply dial tone to the line. The call 
request delay interval is assumed to 
correspond to the period at the 
beginning of a call attempt during 
which the exchange is unable to 
receive any call address information 
from the subscriber. 


Alerting sending 


For calls terminating on ANALOGUE 
SUBSCRIBER LINES, alerting sending 
delay is defined as the interval from the 
instant when the last digit is available 
for processing in the exchange until the 
ringing tone is sent backwards toward 
the calling user. 


Call set up delay 


Call set-up delay is defined as the 
interval from the instant when the 
signalling information required for 
routing is received from the incoming 
signalling system until the instant when 
the corresponding signalling 
information is passed to the outgoing 
signalling system. 


Through-connection delay 


Through-connection delay. 


Connection release delay 


Connection release delay is defined as 
the interval from the instant when 
DISCONNECT or RELEASE message 
is received from a signalling system 
until the instant when the connection is 
no longer available for use on the call 
(and is available for use on another 
call) and a corresponding RELEASE or 
DISCONNECT message is passed to 
the other signalling system involved in 
the connection. 


Speech quality analysis 




Speech Quality - PESQ 


PESQ (ITU-T Rec. P.862 [i.3]) 
and ITU-T Rec. P.862. 1 [i.4]. 


Speech Level - Active Level 


ITU-T Rec. P.56 [i.5] method B. 


Speech Level - Peal< 




Speech Level - Noise 




Speech Level - Signal to Interval Noise 
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